Method and apparatus for representing and searching for color images

ABSTRACT

A method of representing a color image comprises selecting a region of the image, selecting one or more colors as representative colors for the region and, for a region having two or more representative colors, calculating two parameters for each representative color including the variance of the color distribution of the image and the color space value and using these parameters to derive descriptors for the image region.

This application is the national phase under 35 U.S.C. § 371 of PCTInternational Application No. PCT/GB00/01667 which has an Internationalfiling date of Apr. 28, 2000, which designated the United States ofAmerica and was published in English.

The present invention relates to a method and apparatus for representinga colour image or a region of an image for searching purposes, and amethod and apparatus for searching for colour images or image regions.

Searching techniques based on image content for retrieving still imagesand video from, for example, multimedia databases are known. Variousimage features, including colour, texture; edge information, shape andmotion, have been used for such techniques. Applications of suchtechniques include Internet search engines, interactive TV, telemedicineand teleshopping.

For the purposes of retrieval of images from an image database, imagesor regions of images are represented by descriptors, includingdescriptors based on colours within the image. Various different typesof colour-based descriptors are known, including the average colour ofan image region, statistical moments based on colour variation within animage region, a representative colour, such as the colour that coversthe largest area of an image region, and colour histograms, where ahistogram is derived for an image region by counting the number ofpixels in the region of each of a set of predetermined colours.

A known content-based image retrieval system is QBIC (query by imagecontent) (see U.S. Pat. No. 5,579,471, MPEG document M4582/P165: ColourDescriptors for MPEG-7 by IBM Almaden Research Center). In one of themodes of operation of that system, each image in a database is dividedinto blocks. Each block is grouped into subsets of similar colours andthe largest such subset is selected. The average colour of the selectedsubset is chosen as the representative colour of the respective block.The representative colour information for the image is stored in thedatabase. A query in the database can be made by selecting a queryimage. Representative colour information for the query image is derivedin the same manner as described above. The query information is thencompared with the information for the images stored in the databaseusing an algorithm to locate the closest matches.

MPEG document M4582/P437 and U.S. Pat. No. 5,586,197 disclose a similarapproach, but using a more flexible method of dividing an image intoblocks and a different method of comparing images. In another variation,described in MPEG document M4582/P576: Colour representation for visualobjects, a single value for each of two representative colours perregion are used.

Several techniques for representing images based on colour histogramshave been developed, such as MPEG document M4582/P76: A colourdescriptor for MPEG-7: Variable-Bin colour histogram. Other techniquesuse statistical descriptions of the colour distribution in an imageregion. For example, MPEG document M4582/P549: Colour Descriptor byusing picture information measure of subregions in video sequencesdiscloses a technique whereby an image is divided into high and lowentropy regions and colour distribution features are calculated for eachtype of region. MPEG document M4852/P319: MPEG-7 Colour DescriptorProposal describes using a mean and a covariance value as descriptorsfor an image region.

All the approaches described above have important shortcomings. Some ofthem, in particular colour histogram techniques, are highly accurate,but require relatively large amounts of storage and processing time.Other methods, such as the ones using one or two representative colours,have high storage and computational efficiency but are not preciseenough. The statistical descriptors are a compromise between those twotypes of techniques, but they can suffer from lack of flexibility,especially in case where colours of pixels vary widely within a region.

The present invention provides a method of representing an image byapproximating the colour distribution using a number of componentdistributions, each corresponding to a representative colour in an imageregion, to derive descriptors of the image region.

The invention also provides a method of searching for images using suchdescriptors.

The invention also provides a computer program for implementing saidmethods and a computer-readable medium storing such a computer program.The computer-readable medium may be a separable medium such as a floppydisc or CD-ROM or memory such as RAM.

An embodiment of the invention will be described with reference to theaccompanying drawings of which:

FIG. 1 is a block diagram of a system according to an embodiment of theinvention;

FIG. 2 is a flow chart of a first search method; and

FIG. 3 is a flow chart of a second search method.

A system according to an embodiment of the invention is shown in FIG. 1.The system includes a control unit 2 such as a computer for controllingoperation of the system, a display unit 4 such as a monitor, connectedto the control unit 2, for displaying outputs including images and textand a pointing device 6 such as a mouse for inputting instructions tothe control unit 2. The system also includes an image database 8 storingdigital versions of a plurality of images and a descriptor database 10storing descriptor information, described in more detail below, for eachof the images stored in the image database 8. Each of the image database8 and the descriptor database 10 is connected to the control unit 2. Thesystem also includes a search engine 12 which is a computer programunder the control of the control unit 2 and which operates on thedescriptor database 10.

In this embodiment, the elements of the system are provided on a singlesite, such as an image library, where the components of the system arepermanently linked.

The descriptor database 10 stores descriptors of all the images storedin the image database. More specifically, in this embodiment, thedescriptor database 10 contains descriptors for each of a plurality ofregions of each image. The descriptors are derived as described below.

Each image in the database 8 is divided into a number of non-overlappingrectangular blocks of pixels. For each block, a colour histogram is thenderived, by selecting a predetermined number of colours, and countingthe number of pixels in the block of each colour.

The colour histogram so obtained shows the colour distribution of thepixels within the block. In general, the region will have one or moredominant colours, and the histogram will have peaks corresponding tothose colours.

The descriptors for the blocks are based on the dominant colours asidentified from the histogram. The descriptor for each block has thefollowing elements:

-   -   (1) The number of dominant colours, n, called the degree of the        descriptor, where n>1; and    -   for each dominant colour:    -   (2)(a) a weight representing the relative significance of the        respective dominant colour in the block. Here, the weight is a        ratio of the number of pixels in the block of the relevant        colour to the total number of pixels in the block.        $\text{a~~mean~~value,~~m} = \begin{pmatrix}        m_{x} \\        m_{y} \\        m_{z}        \end{pmatrix}$        where x, y and z index colour components, for example the red,        green and blue colour components of the colour in RGB colour        space. Here, the mean value corresponds to the colour components        of the respective dominant colour.        $\text{a~~covariance~~matrix~~C} = \begin{pmatrix}        c_{xx} & c_{xy} & c_{xz} \\        c_{yx} & c_{yy} & c_{yz} \\        c_{zx} & c_{zy} & c_{zz}        \end{pmatrix}$        where c_(ii) represents variance of colour component i and        c_(ij) represents covariance between components i and j. The        covariance matrix is symmetrical (c_(ij)=c_(ji)) so only six        numbers are needed to store it.

In obtaining the descriptor as discussed above, the colour distributionis treated as n different sub-distributions, where n is the number ofdominant colours, each sub-distribution centring about a respectivedominant colour as the mean. The ranges of the sub-distributions maywell overlap, and a suitable algorithm is used to determine the range ofeach distribution for calculating the weight, mean and covariancematrix, as will be understood by a person skilled in the art. One way ofestimating the descriptor components is to fit Gaussian functionscentred at histogram peaks to the histogram by minimising the differencebetween the actual histogram counts and values estimated from themixture of Gaussian functions.

The descriptor database 10 stores a descriptor as defined above for eachblock of each image stored in the image database 8. The representationof the colour distribution within each block using the descriptorstructure described above contains a large amount of descriptiveinformation, but requires less storage space than, for example, fullhistogram information.

As an example, a colour histogram for a specific block may exhibit threepeaks corresponding to three dominant colours. The histogram colourdistribution is analysed as three colour sub-distributions and resultsin a descriptor including the number three indicating the number ofdominant colours, three weights, three mean vectors, corresponding tothe colour vectors for the three peaks, and three correspondingcovariance matrices.

The system is used to search for images in the image database using thedescriptors stored in the descriptor database. The present embodimentprovides two search methods: a single colour based search and a regionbased search.

The single colour based search will be described with reference to theflowchart shown in FIG. 2.

In the single colour based search, the user inputs a query by selectinga colour to be searched, using the pointing device 6 and a menu such asa colour wheel or a palette displayed on the display unit 4 (step 102).The control unit 2 then obtains the corresponding colour vector for thequery colour, the colour vector having components which are therespective colour components for the query colour, that is, the red,green and blue components (step 104).

The control unit 2 then uses the search engine 12 to search for imagesin the image database 8 that include the query colour. The search engine12 performs a matching procedure using the query colour vector and thedescriptors for the image blocks in the descriptor database 10 (step106).

The matching procedure is performed using the following formula forcalculating a matching value M.$M = {\exp\mspace{14mu}\lbrack {{- \frac{1}{2}}( {q - m} )^{T}{C^{- 1}( {q - m} )}} \rbrack}$where q is the query colour vector. A matching value is calculated foreach dominant colour in each block using each value of m and C in thedescriptor for the block. Thus, for a descriptor of degree n, n matchingvalues are obtained.

The matching value can be considered as the value of the probabilitydensity function corresponding to each colour sub-distribution in theblock at the point defined by the query colour value, modelling theprobability density function as a Gaussian function.

For a given descriptor, the larger a matching value M, the closer thecorresponding block is to a match with the selected colour.

When matching values have been calculated for each descriptor in thedescriptor database 10, the search engine 12 orders the results by thesize of M starting with the largest values of M, considering only thelargest value of M for any descriptors of degree greater than one (step108).

The control unit 2 takes the results of the matching procedure from thesearch engine 12, and retrieves from the image database a predeterminednumber K of those images which are the closest matches, corresponding tothe K highest values of M. Those images are then displayed on thedisplay unit 4 (step 110). The set-up of the control unit 2 determineshow many of the closest matches are to be displayed on the display unit.That number can be changed by the user.

As will be understood from the above description, the single colourbased search retrieves images from the image database 8 which have ablock which has a dominant colour which is the same as or close to thecolour initially selected by the user.

The region based search will be described with reference to theflowchart shown in FIG. 3.

In the region based search, the control unit 2 operates to display apredetermined set of search images, which are images from the imagedatabase 8, on the display unit 4 (step 202). The search images may bewholly determined by the set-up of the control unit, or may depend onother requirements input by the user. For example, in a larger systemsupporting keyword-based searches the user might input the word “leaves”which would result in a predetermined set of images depicting leavesbeing shown as the images for colour based search.

Each of the search images is shown with a grid dividing the image intoblocks, corresponding to blocks for which the descriptors have beenderived. The user then selects, using the pointing device 6, a block onone of the images which shows a colour distribution of interest (step204).

The control unit 2 then retrieves the descriptor for the selected imageblock from the descriptor database 10 and uses it as a query descriptor(step 206). The descriptor is already available because the searchimages are taken from the image database 8. The search engine thenperforms a search comparing that query descriptor with the otherdescriptors stored in the descriptor database using matching functions(step 208).

For a query descriptor having a mean value m_(a) and covariance matrixC_(a) for one of the dominant colours and another descriptor having amean value m_(b) and covariance matrix C_(b) for one of the dominantcolours, a matching function is defined as:${m_{s}( {a,b} )} = {\int{{\exp\lbrack {{- \frac{1}{2}}( {q - m_{a}} )^{T}{C_{2}^{- 1}( {q - m_{a}} )}} \rbrack}{\exp\lbrack {{- \frac{1}{2}}( {q - m_{b}} )^{T}{C_{b}^{- 1}( {q - m} )}} \rbrack}d\; q}}$where q is a 3-d vector akin to a colour vector and where the integralis calculated over the range from (0, 0, 0) to (255, 255, 255) where 255is the maximum value of a colour component. The range of the integral inother embodiments will depend upon the colour co-ordinate system andrepresentation used.

This is equivalent to modelling the corresponding coloursub-distributions for the image blocks as probability mass functions inthe form of Gaussian functions, and determining the degree to which theyoverlap, or in other words determining the similarity between them. Thelarger the result of the above calculation, the closer are thecorresponding colour distributions. In this case, the functiondetermines the degree to which a colour sub-distribution in the queryimage block and a colour sub-distribution in a stored image overlap.

The full matching function for matching one descriptor with another isdefined as:$m_{f} = {\sum\limits_{i,j}{v_{i}w_{j}{m_{s}( {i,j} )}}}$where v and w are weights for sub-distributions, and the summation isover all sub-distributions in both regions.

Thus, for each dominant colour described in the descriptor of a queryimage block, a matching value is calculated with respect to eachdominant colour in a descriptor from the descriptor database 10. Theresulting matching values are weighted and then summed to give a finalmatching value corresponding to m_(f).

Full matching values are calculated as described above for alldescriptors in the database with respect to the query descriptor. As inthe single colour based search, the results are ordered (step 210), andthe K images with the highest matching values, indicating the closestmatches, are displayed on the display unit for the user (step 212).

A further iteration of a search can be performed by selecting an imageregion in an image found in the previous search.

Matching can be done using other similarity measures than thosedescribed above. A further example is given below.

For a pair of descriptors, F₁ and F₂, for two regions, a similaritymeasure D is defined as follows:${{D( {F_{1},F_{2}} )} = {{\sum\limits_{j = 1}^{N_{1}}{\sum\limits_{i = 1}^{N_{1}}{p_{1i}p_{1j}f_{1{i1j}}}}} + {\sum\limits_{j = 1}^{N_{1}}{\sum\limits_{i = 1}^{N_{1}}{p_{2i}p_{2j}f_{2{i2j}}}}} - {\sum\limits_{j = 1}^{N_{1}}{\sum\limits_{i = 1}^{N_{1}}{p_{1i}p_{2j}f_{1{i2j}}}}}}},$where$f_{xiyj} = {\frac{1}{2\pi\sqrt{v_{xiyjl}v_{xiyju}v_{xiyjv}}}{\exp\lbrack {{- ( {\frac{c_{xiyjl}}{v_{xiyjl}} + \frac{c_{xiyju}}{v_{xiyju}} + \frac{c_{xiyjv}}{v_{xiyjv}}} )}/2} \rbrack}}$andc _(xiyjl)=(c _(xil) −c _(yjl))² , v _(xiyjl)=(v _(xil) +v _(yjl)),c _(xiyju)=(c _(xiu) −c _(yju))² , v _(xiyju)=(v _(xiu) +v _(yju)),c _(xiyjv)=(c _(xiv) −c _(yjv))² , v _(xiyjv)=(v _(xiv) +v _(yjv)).Here, i and j index the representative colours;

-   -   x and y index the descriptors;    -   N₁ is the number of representative colours in the first        descriptor;    -   N₂ is the number of representative colours in the second        descriptor;    -   p_(1i) is the ith weight in the first descriptor,    -   p_(2j) is the jth weight in the second descriptor;    -   l, u and v represent colour components, such as red, green and        blue colour components in this specific example; and    -   c and v are the dominant colour values (mean values) and colour        variances respectively, so c_(xil) is the lth component of the        ith representative colour value of the xth descriptor, and        v_(xil) is the lth component of the variance of the ith        representative colour of the xth descriptor etc.

In contrast with the matching functions described previously, fordescriptors F₁ and F₁, the smaller the value of D, the closer is thematch between the regions corresponding to the descriptors F₁ and F₂.Accordingly, the values D resulting from a search procedure such asdescribed above are ordered in increasing size starting with thesmallest value of D. Otherwise, the searching and matching procedure canbe carried out substantially as described above, with appropriatemodifications to take account of the different similarity measure. Itwill be noted that this similarity measure uses the variances, and notthe covariance matrix. Thus, the descriptor for a region includes thevariances but does not need the covariance matrix. Accordingly, thestorage requirement is reduced compared with the descriptor describedpreviously.

A system according to the invention may, for example, be provided in animage library. Alternatively, the databases may be sited remote from thecontrol unit of the system, connected to the control unit by a temporarylink such as a telephone line or by a network such as the Internet. Theimage and descriptor databases may be provided, for example, inpermanent storage or on portable data storage media such as CD-ROMs orDVDs.

In the above description, the colour representations have been describedin terms of red, green and blue colour components. Of course, otherrepresentations can be used, such as a representation using a hue,saturation and intensity, or YUV co-ordinate system, or a subset ofcolour components in any colour space, for example only hue andsaturation in HSI.

The embodiment of the invention described above uses descriptors derivedfor rectangular blocks of images. Other sub-regions of the image couldbe used as the basis for the descriptors. For example, regions ofdifferent shapes and sizes could be used. Alternatively, descriptors maybe derived for regions of the image corresponding to objects, forexample, a car, a house or a person. In either case, descriptors may bederived for all of the image or only part of it.

In the search procedure, instead of inputting a simple colour query orselecting an image block, the user can, for example, use the pointingdevice to describe a region of an image, say, by encircling it,whereupon the control unit derives a descriptor for that region and usesit for searching in a similar manner as described above. Also, insteadof using images already stored in the image database for initiating asearch, an image could be input into the system using, for example, animage scanner or a digital camera. In order to perform a search in sucha situation, again the system first derives descriptors for the image orregions of the image, either automatically or as determined by the user.

Appropriate aspects of the invention can be implemented using hardwareor software.

In the above embodiments, the component sub-distributions for eachrepresentative colour are approximated using Gaussian functions, and themean and covariances of those functions are used as descriptor values.However, other functions or parameters can be used to approximate thecomponent distributions, for example, using basis functions such as sineand cosine, with descriptors based on those functions.

1. A method of representing a colour image comprising selecting two ormore colours as dominant colours for an image and, for each dominantcolour, calculating variance of colour distribution of the image incolour space relative to the dominant colour, and representing the imagein terms of colour space value of each dominant colour and value ofrespective colour variance; and forming a descriptor including thecolour space value of each dominant colour and the respective colourvariance value.
 2. A method as claimed in claim 1 wherein the saidselecting includes deriving a colour histogram for a region of the imageincluding the dominant colours.
 3. A method as claimed in claim 2wherein said selecting includes identifying local peaks in the colourhistogram and selecting corresponding colours as dominant colours.
 4. Amethod as claimed in claim 1 wherein said selecting includes selecting aregion of the image, wherein the dominant colours are selected for theimage region.
 5. A method as claimed in claim 4 wherein said selectingthe image region is independent of image content.
 6. A method as claimedin claim 5 wherein said selecting includes selecting the image regionwhich is a polygon.
 7. A method as claimed in claim 4 wherein saidselecting includes selecting the image region which corresponds to anobject.
 8. A method as claimed in claim 1 wherein said representingincludes representing the image in terms of colour space value of eachdominant colour, wherein the colour space value being expressed in termsof a plurality of colour components, and wherein said calculatingincludes calculating the colour variance for each colour component.
 9. Amethod as claimed in claim 8 wherein said calculating includescalculating colour covariances for pairs of colour components.
 10. Anapparatus for implementing a method according to claim
 1. 11. A computersystem programmed to operate according to a method as claimed inclaim
 1. 12. A computer program for implementing a method as claimed inclaim
 1. 13. A computer-readable medium storing computer-executableprocess steps for implementing a method as claimed in claim
 1. 14. Amethod as claimed in claim 1 wherein the descriptor additionallyincludes one or more elements indicating number of dominant colours,related covariance values, and weights indicating the relativesignificance of the dominant colour in the image region.
 15. A method asclaimed in claim 1 comprising storing said descriptor in a data storagemeans.